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PROBLEMS ASSOCIATED V T TH RELATING 
NORMATIVE AND SKILL TESTS 

Sanford Temkin 

Research for Better Schools, Inc. 

(Talk given at University of Pittsburgh on November 3, 1967 to the 
Pennsylvania Educational Research Association.) 

Research for Better Schools (RBS) has been involved in the field 
testing of the Individually Prescribed Instruction (IPI) Project vjhich 
has been developed by the Learning Research and Development Center (LRDC) 
of the University of Pittsburgh. A part of our evaluation efforts has 
been focused on two different kinds of tests — the IPI Placement Test 

and the Iowa Test of Basic Skills (ITBS) , Form 4 • 

• . 

Before moving into the main discussion it is necessary to examine 
the structure of these tests. The IPI Placement Tests are given to each 
child in the program. This program is built around a set of behaviorally 
defined objectives. The placement testing is designed to contain a sam- 
pling of ite ms which will predict performance on these objectives. The 
test is divided by Level (B, C, ..., G). Each of these Levels contains 
sub-tests of the various Mathematics Areas which comprise the IPI curricu- 
lum.* The Placement Test generates, as an output, a Placement Profile 




* These Mathematics Areas arc: Numeration, Place Value,- Addition, Sub- 

traction, Multiplication, Division, Combination of Processes, Fractions, 
Money, Time, Systems of Measurement, Geometry, Special Topics, and 
Supplementary Topics. 
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which is a set of very gross starting levels. This allows the teacher 

ft 

to start the child at those points in the continuum "where he is". 

The ITBS is an achievement test, although the test does have 
sub-tests which have gross "skill" outputs (e.g. Spelling, Punctuation). 
This test was selected because of its wide usage. It was also felt 
that this test had "more in common" with the IPI curriculum than any 
of the other widely used standardized tests. RBS decided to give four 
sub-tests of the ITBS battery. These are: 

Arithmetic Concepts 
Arithmetic Problem Solving 
Vocabulary 
Reading 

The arithmetic sub-tests were given on an untimed -ungraded basis 
that is - each child (grades 3-6) started at the first item on the third 
grade test and worked as far as he was able. The Vocabulary and Reading 

i 

i 

Tests were given under the usual time and grade constraints. 

With this much of an introduction we shall look at three kinds 
of specific problems: 

Problem I: Description - i.e. How can the IPI Placement 

Test be described to the non-IPI community - 
how does it look, feel, taste? 

Problem II: Pre-Post Comparisons - i.e. What differences, 

if any, can be seen after a short interval of 
time has elapsed? 



* 




Appendix A contains an exar.nl e of a child f s Placement Profile. 



3 



0 



Problem III: Curriculum Comment - i.e. What information 

can be obtained relative to the IPI Curriculum? 

Problem I: Description 

If we were to describe pupil placement strictly in IPI terms, 
many interested persons would not have a framework which is satisfactory 
for purposes of appreciating pupil performance outcomes. If, however, 
we attempt to describe the IPI Placement Test results in terms of ITBS 
performances we may be able to assist the "outside" educational commu- 
nity. The following is one way to describe the initial performance of 
two groups of children (IPI and control) in terms of the two tests (IPI 
Placement and ITBS): 

First let us look at some description of performances 
on the ITBS. It should be emphasized that the objective is 
description and not a basis for quantitative substitution 
into a mathematical model. 

1. Consider each grade level as a population. 

2. Consider the ITBS Reading Comprehension score 
as a basis of stratification (classification). 

3. Compute the means and standard deviations by school 
and for all schools (IPI versus control). These 
calculations will treat each of the two ITBS 




arithmetic tests as a separate problem. 



TABLE 



• • • 
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ITBS ARITHMETIC CONCEPTS BY READING COMPREHENSION 
GROUP (Grade 3) 






SCHOOL 




0 



ERIC 



5 



0 



This table provides a description of control and IPI schools 
for 3rd grade children performances on the ITBS. We will be able to 
determine the extent to which reading levels condition performance on 
a school by school basis in the ITBS Arithmetic Tests. 

Now we can proceed to the IPI Placement Test description. 

1. Consider each grade level as a population. 

2* Consider each Mathematics Area (e.g. Numeration) 
as a separate problem. 

3* Consider the ITBS Reading Comprehension score as 
a basis of stratification. 

4. Compute the median ITBS score and the "interquartile 
range for the particular grade level. 

5. Relate the median performance on ITBS to the median 
performance on the IPI Placement Test. Try the 
same for the interquartile range. 

Table € 2 provides a description of the relationships between ITBS 
medians and the median placement in each Mathematics Area by school. 
Reading level is also taken into consideration. 
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ITBS ARITHMETIC CONCEPT MEDIANS BY READING COMPREHENSION 
GROUr WITH IPI PLACEMENT MEDIANS (Grade 3) 



SCHOOL 



Reading Comprehension 


AA 


AB 


• • • 


High 

Median ITBS 

Median Placement 

Numeration 
Place Value 
Addition 
• 

• 

• 

Middle 

Median ITBS 

Median Placement 

Numeration 
Place Value 
Addition 
• 

• 

• 

Low 

Median ITBS 

Median Placement 

Numeration 
Place Value 
Addition 
• 

• 

• 




• 


• 


All 

Pupils 

Median ITBS 

Median Placement 

Numeration 
Place Value 
Addition 
• 

• 

• 

i 


• 
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IPI PLACEMENT LEVELS WITH MEDIAN ITBS ARITHMETIC 
SUB-TEST SCORES (Grade 3: Numeration) 



IPI Level 

B 

C 

D 

E 

F 

G 

H 



Arithmetic ( ITBS ) 
Problem Solving (Median) 



Arithmetic ( ITBS ) 
Concepts (Median) 



ERJC 



75 
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This information would add descriptive power in that it will 
show the relationship between the respective placement levels in IPI 
and the ITBS median scores. 

Problem II: Pre-Post Comparisons 

The pre-post problem focuses on the need to make assessments 
of pupil performance over time. The pre-test portions of both tests 
were given in September and October of 1967. There were substantial 
differences in the starting times as well as some .reversals in the 
order of testing (we had indicated that the ITBS should be given 
immediately after the administration of IPI Placement Tests). A 
confounding factor is the length of exposure to the IPI Program. This 
varies from school to school and in some cases within school. A way 
to back out of this situation is to treat each school-grade combination 
as distinct. This makes for a "micro-analysis" and will detract from 
overall generalization power. 

The first major question which needs to be answered is - did 
the IPI and control groups start out the same? For the sake of sim- 
plicity we can think of the third grade group for each school. Now 
we need a criterion. The simplest place to start is with the ITBS 
scores. The Analysis of Variance will indicate if we can treat all 
third grade groups as a single group with respect to averages. If not, 
we can create some sub-groups for subsequent analysis. Once these sub- 
groups have been structured we can proceed to an implicit question - 
how do these sub-groups compare on the IPI Placement Test? In this 
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instance we can use the chi-squared (x 2 ) test. This procedure will 

* 

generate four kinds of sub-groups. . 

1. Those with similar average pre-ITBS scores, only. 

2. Those with similar frequency patterns on pre-IPI 
Placement Testing, only. 

3. Those with similar levels of both of the above. 

4. Those with none of the above: micro-analysis. 

This .leads us to the next major question in this pre-post frame- 
work - how did the respective sub-groups perform over time? The procedure 
to be employed for ITBS scores is a comparison of pre and post means with- 
in each sub-group. The nature of the sub-group composition (i.e. which 
of the above four sub-group types it belongs to) will determine the kinds 
of comparisons to be made. For example, a sub-group whose members have 
only similar pre-ITBS averages will be compared on a pre-post basis only 
with respect to ITBS averages. These school-grade combinations would be 
compared on a micro-analysis basis for IPI Placement pre-post comparisons. 

Problem III: Curriculum Comment 

When a test constructor sets out on his task, he tries out many 
more items than are ultimately used in the test. He discards those items 
which "everyone" gets as well as those items which "no one" gets. Perhaps 
the preceding is not totally relevant but it is fair to say that what 
actually happens is that the test measures, to a great extent, the curriculum 

* Assuming that the analysis of variance and the x 2 tests have rejected 
the- hypotheses of "no significant difference". Appendix B outlines 
the analytic flow of this section. 
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that the students were exposed to at the time of the test development. 
If another curriculum is designed with curriculum objective weights 
which differ substantially from "a standard weighted curriculum" then 

the students using the new curriculum may not do well on the existing 

* 

test. 

This is not meant as an apology but rather it should provide a 
basis to ask some interesting questions. Suppose we look at a child 1 s 
possible joint outcomes on the first item of 1TBS and the first item of 
the IPI Placement Test. 



ITBS 

Item 

#1 



IPI Item #1 





Right 


Wrong 


Right 


• 




Wrong 







Now we can sum the number of tallies in the blocks over all 



children and we will obtain a pattern. If the pattern is random we 
are dealing with unrelated items. If the pattern is RIGHT-RIGHT, then 
"both curricula" handle these types of items. It is clear that what 
is needed is an analysis of the RIGHT-WRONG and WRONG-RIGHT combinations 
of items. The actual framework would appear as a large matrix (Nj pj x 
perhaps in terms of correlation coefficients. The analysis would presum- 
ably be best handled in the hands of curriculum experts. . 



* For instance, an IPI child works on Geometry, Fractions, and Money 
skills at Level B (roughly first grade), and consequently would be 
penalized by tests which arc fair to* most curricula. 
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Concluding Comments : 

It is fair to say that no solution to the three major problems 
discussed today will be totally satisfactory.. A group test cannot 
comment adequately on an individualized program; especially a program 
5 hich is different in content as well as structure. Nevertheless, the 
problems are real and it is hoped that the ITBS data coupled with IPI 
Placement data can provide useful information for descriptive and 
decision making purposes. 
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At rfcnurx it : mow tor answering the question: Old IF! and Control 

Groups Start Out Similar? 




If Yes: If No; 



This output coupled with 
the outputs from above de- 
termine which of the 4 types 
of pre-post analysis will be 
done for each school-grade 
combination. 



